Multi-perspective system and method for behavioral policy selection by an autonomous agent

ABSTRACT

A system and a method for autonomous decisioning and operation by an autonomous agent includes: collecting decisioning data including: collecting a first stream of data includes observation data obtained by onboard sensors of the autonomous agent, wherein each of the onboard sensors is physically arranged on the autonomous agent; collecting a second stream of data includes observation data obtained by offboard infrastructure devices, the offboard infrastructure devices being arranged geographically remote from and in an operating environment of the autonomous agent; implementing a decisioning data buffer that includes the first stream of data from the onboard sensors and the second stream of data from the offboard sensors; generating current state data; generating/estimating intent data for each of one or more agents within the operating environment of the autonomous agent; identifying a plurality of candidate behavioral policies; and selecting and executing at least one of the plurality of candidate behavioral policies.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/705,503, filed 6 Dec. 2019, which is a continuation of U.S. patentapplication Ser. No. 16/514,624, filed 17 Jul. 2019, which claims thebenefit of US Provisional Application No. 62/701,014, filed 20 Jul.2018, both of which are incorporated in their entireties by thisreference.

TECHNICAL FIELD

The inventions relate generally to the vehicle automation field, andmore specifically to new and useful systems and methods for selectingbehavioral policy by an autonomous agent.

BACKGROUND

State of the art vehicle automation presently enables some vehicles,such as cars, to operate in a substantially and/or sometimes fullyautonomous state. An ability of such autonomous agents to operateeffectively and safely in busy or active environments often relies on anability of the autonomous agent to observe its operating environment andmake operating decisions that enables the autonomous agent to achieve arouting or traveling goal in a safe manner.

A technical problem that may arise in many operating circumstancesinvolving an autonomous agent may relate to an inability of theautonomous agent to select or make the most optimal operating decisionwhen multiple operating decisions are possible in a given operatingcircumstance. While route planning and low-level control instructionsmay provide a basis for performing self-controls by an autonomous agentfor achieving a particular destination, behavioral planning typicallyprovides a basis for performing real-time decisions by the autonomousagent according to live observations of the operating environment madeby one or more sensors onboard the autonomous agent. In particular, theautonomous agent's perspective of its real-time environment forselecting behavioral policy is primarily shaped by onboard sensorscarried by the autonomous agent. Thus, the technical problem persistsbecause the autonomous agent may have only a singular view of itsoperating environment rather than comprehensive perspectives of theoperating environment to enable an optimal selection of behavioralpolicy in real-time operating circumstances.

Thus, there is a need in the vehicle automation field for enabling amulti-perspective view of an operating environment of an autonomousagent that enables an optimal selection of behavioral policy by anautonomous agent in real-time operating conditions. The embodiments ofthe present application described herein provide technical solutionsthat address, at least, the need described above.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 illustrates a schematic representation of a system forimplementing an autonomous agent in accordance with one or moreembodiments of the present application;

FIG. 1A illustrates an example schematic representation of an autonomousagent operating system in accordance with one or more embodiments of thepresent application;

FIG. 2 illustrates an example method in accordance with one or moreembodiments of the present application;

FIG. 3 illustrates an example schematic with prospective behavioralpolicies for an autonomous agent in accordance with one or moreembodiments of the present application; and

FIG. 4 illustrates an example schematic implementing informativeinfrastructure data for behavioral policy selection by an autonomousagent in accordance with one or more embodiments of the presentapplication.

BRIEF DESCRIPTION OF THE INVENTION(S)

In one embodiment, a system for intelligently implementing an autonomousagent includes: a plurality of offboard infrastructure devices arrangedgeographically remote an autonomous agent and that: collect observationdata relating to circumstances surrounding a travel route of theautonomous agent; the autonomous agent comprising: a communicationinterface that enables the autonomous agent to communicate with each ofthe plurality of offboard infrastructure devices; an onboard sensorsuite comprising a plurality of distinct sensors arranged on theautonomous agent and that collect observation data relating tocircumstances surrounding the autonomous agent from a perspective thatis distinct a perspective of the plurality of offboard infrastructuredevices; a decisioning data buffer storing at least a first stream ofobservation data from the onboard sensor suite and a second stream ofobservation data from the plurality of offboard infrastructure devices;one or more onboard computing devices that: implements a tracking of oneor more agents within the circumstances surrounding the autonomous agentbased on the first stream of data and the second stream of data;computes an intent estimation for each of the one or more agents basedon the first stream of data and the second stream of data; computesstate data for each of the one or more agents and for the autonomousagent; implements a multi-policy decision-making module that identifiesa plurality of candidate behavioral policies for the autonomous agentbased on the tracking and the intent estimation; selects one of theplurality of candidate behavioral policies and executes the selected oneof the plurality of candidate behavioral policies for controlling anautonomous operation of the autonomous agent.

In one embodiment, the one or more onboard computing devices furtherimplement: a time synchronization module that synchronizes the firststream of observation data and the second stream of observation data tothe autonomous agent by synchronizing: the first stream of observationdata to the autonomous agent based on a first computed communicationlatency between the onboard sensor suite and the autonomous agent; thesecond stream of observation data to the autonomous agent based on asecond computed communication latency between the plurality of offboardinfrastructure devices and the autonomous agent.

In one embodiment, the one or more onboard computing devices further:rearrange a position of the first stream of observation data from afirst position to a second position within the decisioning data bufferbased on the synchronization; and rearrange a position of the secondstream of observation data from a first position to a second positionwithin the decisioning data buffer based on the synchronization.

In one embodiment, the one or more onboard computing devices furtherimplement: a synchronization module that synchronizes the first streamof observation data of the autonomous agent and the second stream ofobservation data from offboard infrastructure devices according to acommon clock of the autonomous agent.

In one embodiment, the one or more onboard computing devices: inresponse to the synchronization of the first stream of observation dataand the second stream of observation data, data from the first stream ofobservation data and data from the second stream of observation data arerepositioned to a historical position within the decisioning data bufferthat is associated with an earlier point in time relative to data thatis positioned beyond the data from the first stream and the secondstream of observation data.

In one embodiment, the one or more onboard computing devices: storesdata obtained from each of the plurality of offboard infrastructuredevices and data obtained from each of the plurality of distinct sensorsof the onboard sensor suite within a distinct track of memory that isindependent from other tracks of memory, wherein an intent estimationfor any agents identified within the data obtained from each of theplurality of offboard infrastructure devices and the data obtained fromeach of the plurality of distinct sensors is computed based on thedistinct track of memory for the respective offboard infrastructuredevice or the respective distinct sensor.

In one embodiment, the distinct track of memory for the data obtainedfrom each of the plurality of offboard infrastructure devices and thedata obtained from each of the plurality of distinct sensors is combinedinto a master track of memory; the intent estimation for each of the oneor more agents is based on the master track of memory.

In one embodiment, the system may include a remote autonomous agentservice being implemented by a distributed network of computing devicesand that is in operable communication with each of the autonomous agentand each of the plurality of offboard infrastructure devices, whereinthe remote autonomous agent service computes one or more: the trackingof one or more agents within the circumstances surrounding theautonomous agent based on the first stream of data and the second streamof data; the intent estimation for each of the one or more agents basedon the first stream of data and the second stream of data.

In one embodiment, a method for autonomous decisioning and operation byan autonomous agent includes: collecting decisioning data including:collecting a first stream of data comprising observation data obtainedby one or more onboard sensors of the autonomous agent, wherein each ofthe one or more onboard sensors is physically arranged on the autonomousagent; collecting a second stream of data comprising observation dataobtained by one or more offboard infrastructure devices, the one or moreoffboard infrastructure devices being arranged geographically remotefrom and in an operating environment of the autonomous agent;implementing a decisioning data buffer that includes the first stream ofdata from the one or more onboard sensors and the second stream of datafrom the offboard sensors; generating current state data;generating/estimating intent data for each of one or more agents withinthe operating environment of the autonomous agent; identifying aplurality of candidate behavioral policies; and selecting and executingat least one of the plurality of candidate behavioral policies.

In one embodiment, the operating environment of the autonomous agentincluding a predetermined radius from a geographical position of theautonomous agent while operating along a structured or an unstructuredroute of the autonomous agent.

In one embodiment, the first stream of data includes data relating tosensed observations of circumstances surrounding the autonomous agentobtained from a perspective of the autonomous agent by each of the oneor more onboard sensors; the second stream of data includes datarelating to sensed observations of circumstances within the operatingenvironment of the autonomous agent obtained from an externalperspective toward a route of the autonomous agent that is made by eachof the offboard infrastructure devices.

In one embodiment, the second stream of data includes data relating toan operating state of at least one of the one or more offboardinfrastructure devices.

In one embodiment, in response to traveling into a communicationproximity of at least one of the one or more offboard infrastructuredevices, automatically collecting the second stream of data from the atleast one or more offboard infrastructure devices within thecommunication proximity.

In one embodiment, a field-of-sensing of the one or more offboardinfrastructure devices comprises a geometrically defined region, the oneor more offboard infrastructure devices may be configured to sense orcollect semantic abstractions of objects within the geometricallydefined region.

In one embodiment, the second stream of data from the offboardinfrastructure devices includes semantically dense state data of ascene.

In one embodiment, implementing the decisioning data buffer includes:sequentially storing data received from the first stream of data and thesecond stream of data based on a time at which each of the first streamof data and the second stream of data was received.

In one embodiment, implementing the decisioning data buffer furtherincludes: computing a global time synchronization between the firststream of data and the second stream of data based on timestamp dataappended with the second stream of data provided by the one or moreoffboard infrastructure device, wherein computing the global timesynchronization includes computing a latency value for the second streamof data based on calculating a difference between a first timestamp dataindicating a first time at which the second stream of data was obtainedby the one or more offboard infrastructure devices and a secondtimestamp data indicating a second time at which the second stream ofdata was collected by the autonomous agent.

In one embodiment, implementing the decisioning data buffer includes:repositioning within the decisioning data buffer data from the seconddata stream based on the computed latency value, wherein therepositioning includes moving the data from the second data stream froma first position to a second position within the decisioning data bufferthat is earlier in historical time.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The following description of the preferred embodiments of the presentapplication are not intended to limit the inventions to these preferredembodiments, but rather to enable any person skilled in the art to makeand use these inventions.

1. Overview

As discussed in the background section, an ability of an autonomousagent to select optimal behavioral policy for navigating real-timeoperating circumstances is lacking due to a limited comprehension of anoperating environment of the autonomous agent. Specifically, modernautonomous agents may use only a single perspective (i.e., theautonomous agent's perspective) for behavioral policy decisioning andfail to take into account perspectives outside of their own that mayenhance a quality of behavioral policy selection. The technical issue ofsuboptimal behavioral policy selection by an autonomous agent may besignificantly compounded due to exponential behavioral policy optionsthat may populate when an operating environment of the autonomous agentincludes many active or dynamic agents with many possible intents. Thus,the many possible intents of active agents in an operating environmentmay result in seemingly intractable variations of behavioral policies.

A technical solution proposed herein that makes tractable a set ofpossible behavioral policy options available for execution by anautonomous agent includes augmenting onboard perceptual data generatedby the autonomous agent with offboard perceptual data obtained fromoffboard data sources.

The embodiments of the present application provide enhanced systems andmethods that enable improved behavioral policy selection using amulti-perspective approach for an enhanced comprehension of an operatingenvironment by an autonomous agent. That is, one or more embodiments ofthe present application, enable an autonomous agent to assimilate dataand/or perspectives from multiple actors outside of or external to theautonomous agent. Accordingly, the autonomous agent may be able to useas input data for decisioning or selecting a behavioral policy data thatwould otherwise be unavailable or imperceptible to the autonomousvehicle based on the limited field-of-detection of its onboard sensors.

Accordingly, one or more embodiments of the present application providesensing devices that are external (e.g., offboard sensing devices,infrastructure sensing devices, or the like) to the autonomous agent andthat may be arranged in an operating environment of the autonomous agentfor collecting data from various perspectives within the operatingenvironment that may be distinct from the one or more perspectives ofthe onboard sensor suite of the autonomous agent. In such embodiments,these offboard sensing devices may function to provide one or morestreams of sensed data to the autonomous agent that may be used asadditional decisioning input data for selecting behavioral policy by theautonomous agent. That is, the one or more embodiments of the presentapplication augment onboard sensing data of the autonomous agent withoffboard sensing data to enable a multi-perspective comprehension of anoperating environment of the autonomous agent ahead of making aselection of a behavioral policy in a real-time or near real-timeoperating circumstance.

2. System for Autonomous Decisioning Using Infrastructure Sensing Data

As shown in FIGS. 1-1A, a system 100 that enables multi-policydecisioning by an autonomous agent includes an autonomous agent no, anonboard computing system 115, a plurality of infrastructure devices 120,and a communication interface 130.

The autonomous agent 110 preferably includes an autonomous vehicle nothat is preferably a fully autonomous vehicle, but may additionally oralternatively be any semi-autonomous or fully autonomous vehicle; e.g.,a boat, an unmanned aerial vehicle, a driverless car, etc. Additionally,or alternatively, the autonomous agent no may be a vehicle that switchesbetween a semi-autonomous state and a fully autonomous state (or afully-manned state) and thus, the autonomous agent no may haveattributes of both a semi-autonomous vehicle and a fully autonomousvehicle depending on the state of the autonomous agent no. While someportions of the embodiments of the present application are describedherein as being implemented via an autonomous agent 110 (e.g., anautonomous vehicle (e.g., a driverless car), a semi-autonomous, anunmanned aerial vehicle (e.g., a drone), or the like) it shall be notedthat any suitable computing device (e.g., any edge device includingmobile computing devices, etc.) may be implemented to process sensordata of an autonomous agent no. While it is generally described that theautonomous agent 110 may be an autonomous vehicle, it shall be notedthat the autonomous agent no may be any type of kind of autonomousmachine, autonomous device, autonomous robot, and/or the like.

In a preferred embodiment, the autonomous agent no includes an onboardcomputing system 115 (e.g., a computer integrated with the autonomousagent) or any suitable vehicle system but can additionally oralternatively be decoupled from the autonomous agent no (e.g., a usermobile device operating independent of the autonomous agent).

Additionally, or alternatively, the onboard computing system 115 mayinclude a processing system (e.g., graphical processing unit or GPU,central processing unit or CPU, or any suitable processing circuitry) aswell as memory. The memory can be short term (e.g., volatile,non-volatile, random access memory or RAM, etc.) and/or long term (e.g.,flash memory, hard disk, etc.) memory. As discussed below, theautonomous agent no may additionally include a communication interface130 that includes a wireless communication system (e.g., Wi-Fi,Bluetooth, cellular 3G, cellular 4G, cellular 5G, multiple-inputmultiple-output or MIMO, one or more radios, or any other suitablewireless communication system or protocol), a wired communication system(e.g., modulated powerline data transfer, Ethernet, or any othersuitable wired data communication system or protocol), sensors, and/or adata transfer bus (e.g., CAN, FlexRay). In a preferred embodiment, theonboard computing system 115 may operate to interact with and/oroperably control any or one or more of the identified components ormodules described herein. For instance, the onboard computing system 115may function to implement and/or execute computer instructions forimplementing a multipolicy decisioning module, a synchronization module,and/or the like.

Additionally, or alternatively, the autonomous agent 110 may be inoperable communication with a remote or disparate computing system thatmay include a user device (e.g., a mobile phone, a laptop, etc.), aremote server, a cloud server, or any other suitable local and/ordistributed computing system remote from the vehicle. The remotecomputing system may preferably be connected to one or more systems ofthe autonomous agent through one or more data connections (e.g.,channels), but can alternatively communicate with the vehicle system inany suitable manner.

The onboard computing system 115 preferably functions to control theautonomous agent no and process sensed data from a sensor suite (e.g., acomputer vision system, LIDAR, flash LIDAR, wheel speed sensors, GPS,etc.) of the autonomous agent no and/or other (infrastructure device120) sensors to determine states of the autonomous agent no and/orstates of agents in an operating environment of the autonomous agent no.Based upon the states of the autonomous agent and/or agents in theoperating environment and programmed instructions, the onboard computingsystem 115 preferably modifies or controls behavior of autonomous agent110. Additionally, or alternatively, the onboard computing system 115preferably includes a multi-policy decision-making module 117 thatfunctions to generate behavioral policies and select a behavioral policythat the onboard computing system 115 may function to execute to controla behavior of the autonomous agent 110.

The onboard computing system 115 is preferably a general-purposecomputer adapted for I/O communication with vehicle control systems andsensor systems but may additionally or alternatively be any suitablecomputing device.

Additionally, or alternatively, the onboard computing system 115 ispreferably connected to the Internet via a wireless connection (e.g.,via a cellular link or connection). Additionally, or alternatively, theonboard computing system 115 may be coupled to any number of wireless orwired communication systems.

The infrastructure devices 120 preferably function to observe one ormore aspects and/or features of an environment and collect observationdata relating to the one or more aspects and/or features of theenvironment. In such preferred embodiments, the infrastructure devicesadditionally function to collect data associated with the observationsand transmit the collected data and/or processed derivatives of thecollected data to the autonomous agent 110. In some implementations, theinfrastructure devices may additionally forward the collectedobservations data to an autonomous vehicle service and/or remoteplatform (e.g., implemented via a network of distributed computingsystems or the like) that operates to interactively communicate withand/or control one or more functions of the autonomous agent 110.

In some embodiments, the infrastructure devices 120 may be referred toherein as roadside units. The roadside units preferably include devicesin an immediate and/or close proximity or within short-rangecommunication proximity to an operating position of an autonomous agent110, such as an autonomous car, and may function to collect dataregarding circumstances surrounding the autonomous agent 110 and inareas proximate to a zone of operation of the autonomous agent no. Insome embodiments, the roadside units may include one or more of offboardsensing devices including flash LIDAR, thermal imaging devices (thermalcameras), still or video capturing devices (e.g., image cameras and/orvideo cameras, etc.), global positioning systems, radar systems,microwave systems, inertial measuring units (IMUs), and/or the like.

The infrastructure devices 120 may additionally or alternatively includecomputing capabilities via processing circuitry and a communicationinterface that enables the infrastructure devices 120 to communicatewith an autonomous agent no. The zone of operation of the autonomousagent 110 may be defined as a predefined radius along a structuredand/or unstructured route of the autonomous agent 110. For instance, inthe case of a structured and/or predefined autonomous agent route, theproximate zone of operation of the autonomous agent may be one hundredfeet (100 ft) from or along any point along the structured route. Insuch embodiments, the zone of operation may be defined as some radius orpredefined distance (e.g., 100 ft) at any point along the structuredroute at which the autonomous agent 110 is positioned and/or operating(e.g., driving).

A technical benefit achieved by the implementation of the infrastructuredevices 120 includes an ability to observe circumstances (e.g., aroundcorners, down perpendicular streets, etc.) beyond the observable scopeof the autonomous agent no. That is, at a given instance in timeobservations of one or more aspects of a given environment may be madeby an autonomous agent 110 and observations of one or more differentand/or overlapping aspects of the given environment may be made from adifferent perspective by one or more infrastructure devices 120 arrangedand operating in the given environment. In such embodiments, theperspective of the infrastructure devices 120, including the observationdata therefrom, may be augmented to observation data from theperspective of the autonomous agent 110 to generate a comprehensiveperspective of the operating environment of the autonomous agent no. Inthis way, improved predictions of the operating environment may be madeand consequently, improved behavioral policy decisions may be selectedand/or executed by the autonomous agent no for operating independently(of a human operator) and safely within the operating environment.

As mentioned above, the autonomous agent no may function to augmentand/or fuse data derived by its own onboard sensor suite with theadditional observations by the infrastructure devices 120 (e.g., theroadside units) 120 to improve behavioral policy selection by theautonomous agent no.

Additionally, or alternatively, in various embodiments theinfrastructure devices 120 are able to detect and track any type or kindof agents in an operating environment, such as with a video camera orradar. In such embodiments, an example video camera may function toprovide detection of agents and semantic classification of the agenttype and possible intent of an agent, such as a pedestrian that is aboutto cross a road, or a car that is about to make a left turn, a driverwhich is about to open a car door and exit their vehicle, a bicyclistoperating in a bike lane, and/or the like.

Additionally, or alternatively, other infrastructure devices 120 mayinclude traffic management devices (e.g., traffic sensors, trafficlights, pedestrian lights, etc.) or the like operating in theenvironment that may function to communicate with one or more of theroadside units 120 and/or communicate directly with the autonomous agent110 regarding data collected and/or sensed by the infrastructure device120, regarding an operating state of the infrastructure device 120(e.g., red or green traffic light), and the like. For example, in thecase that the autonomous agent 110 is an autonomous vehicle, a trafficlight may be an infrastructure device 120 in an environment surroundingthe autonomous vehicle that may function to communicate directly to theautonomous vehicle or to a roadside unit that may be in operablecommunication with the autonomous vehicle. In this example, the trafficlight may function to share and/or communicate operating stateinformation, such as a light color that the traffic light is projecting,or other information, such as a timing of the light changes by thetraffic light, and/or the like.

The communication interface 130 preferably enables the autonomous agentno to communicate and/or exchange data with systems, networks, and/ordevices external to the autonomous agent no. Preferably, thecommunication interface 130 enables one or more infrastructure devices120 to communicate directly with the autonomous agent no. Thecommunication interface 130 preferably includes one or more of acellular system (or any suitable long-range communication system),direct short-wave radio, or any other suitable short-range communicationsystem.

In some embodiments, in addition to a powertrain (or othermovement-enabling mechanism), autonomous agent no may include a sensorsuite (e.g., computer vision system, LIDAR, RADAR, wheel speed sensors,GPS, cameras, etc.) or onboard sensors that are in operablecommunication with the onboard computing system 115.

The onboard sensor suite preferably includes sensors used to performautonomous agent operations (such as autonomous driving) and datacapture regarding the circumstances surrounding the autonomous agent 110as well as data capture relating to operations of the autonomous agent110 but may additionally or alternatively include sensors dedicated todetecting maintenance needs of the autonomous agent 110 For example, thesensor suite may include engine diagnostic sensors or an exteriorpressure sensor strip. As another example, the sensor suite may includesensors dedicated to identifying maintenance needs related tocleanliness of autonomous agent interiors; for example, internalcameras, ammonia sensors, methane sensors, alcohol vapor sensors, etc.

In accordance with one or more embodiments, an autonomous operatingsystem may generally include a controller 116 controls autonomousoperations and/or actions of the autonomous agent 110. That is, suitablesoftware and/or hardware components of controller 116 (e.g., processorand computer-readable storage device) are utilized to generate controlsignals for controlling the autonomous agent 110 according to a routinggoal of the autonomous agent 110 and selected behavioral policies of theautonomous agent 110.

Additionally, or alternatively, the autonomous agent 110 includes asensor fusion system 117, a positioning system 118, and a guidancesystem 119. As can be appreciated, in various embodiments, the sensorsmay be organized into any number of systems (e.g., combined, furtherpartitioned, etc.) as the disclosure is not limited to the presentexamples.

In various embodiments, the sensor fusion system 117 synthesizes andprocesses sensor data and together with a multi-policy decisioningmodule or the like predicts the presence, location, classification,and/or path of objects and features of the environment of the autonomousagent 110. In various embodiments, the sensor fusion system 117 mayfunction to incorporate data from multiple sensors and/or data sources,including but not limited to cameras, LIDARS, radars, infrastructuredevices 120, remote data feeds (Internet-based data feeds), and/or anynumber of other types of sensors.

The positioning system 118 processes sensor data along with other datato determine a position (e.g., a local position relative to a map, anexact position relative to lane of a road, vehicle heading, velocity,etc.) of the autonomous agent 110 relative to the environment. Theguidance system 119 processes sensor data along with other data todetermine a path for the vehicle no to follow.

In various embodiments, the controller 116 may function to implementmachine learning techniques to assist the functionality of thecontroller 116, such as feature detection/classification, obstructionmitigation, route traversal, mapping, sensor integration, ground-truthdetermination, and the like.

3. Method for Autonomous Decisioning Using Infrastructure Sensing Data

As shown in FIG. 2, a method 200 for autonomous decision and control byan autonomous agent including collecting decisioning data S210, buildinga data buffer comprising decisioning data S215, generate (real-time)current state data S220, generating intent data for each identifiedagent S230, identifying potential behavioral policies S240, andselecting one of the plurality of behavioral policies S250. The methodoptionally includes executing the selected behavioral policy S255.

The method 200 preferably functions to enable tractable decision-makingfor an autonomous agent by limiting a selection of a behavior by theautonomous agent to a limited set of plausible policies. In one or morepreferred embodiments, the method 200 enables an autonomous agent togather circumstance data (including environmental data) frominfrastructure devices and infer or predict actions of agents operatingin the environment . . . . The observation data provided by the one ormore infrastructure devices may preferably enable the autonomous agentto determine likely outcomes of the current environment for differentbehaviors of the autonomous agent.

S210, which includes collecting decisioning data, functions to collectstreams of data from one or more data sources that may be used as inputfor decisioning by an autonomous agent. Preferably, S210 may function tocollect the streams of data at an autonomous agent, such as anautonomous vehicle or the like. In a preferred embodiment, the one ormore data sources may include devices and/or system of an autonomousagent, sensors mounted (e.g., onboard sensors) on the autonomous agent,and infrastructure devices in a proximity of the autonomous agent. Itshall be noted that while the one or more data sources preferablyinclude devices and/or systems of the autonomous agent, onboard sensors,and infrastructure devices, the one or more data sources mayadditionally or alternatively include one or more remote data feeds(e.g., weather feed, traffic feed, etc.), a remote autonomous agentplatform (e.g., remote servers, cloud servers, etc. for remotelymanaging and/or operating an autonomous agent), and any other suitabledata source accessible to the autonomous agent.

According to one preferred embodiment, S210 may function to collectdecisioning data from infrastructure devices. In such preferredembodiment, S210 functions to collect the decisioning data during anoperation of the autonomous agent but may also function to collect thedecisioning data during periods in which the autonomous agent is not inan active state (e.g., not driving, not in operation, parked or thelike). The infrastructure devices preferably include one or more sensordevices that are intelligently arranged and/or positioned within anenvironment. For instance, the one or more sensor devices may bearranged to collect data that may be assistive for determining and/orgenerating driving/operating (control) instructions for an autonomousagent and also, for decisioning, by an autonomous agent when presentedwith multiple driving and/or operating instructions, which instructionsto execute and which instructions to disregard. Thus, the one or moreinfrastructure sensors may function to collect data in a drivingenvironment, which may include road data, sidewalk data, positions ofstatic and/or dynamic object data (e.g., agent data), traffic data, andthe like along a given route plan or a possible route plan of a givenautonomous agent. In one or more embodiments, the one or moreinfrastructure devices may function to collect observation data from anexternal and/or environmental perspective toward the autonomous agentand/or toward a plan route of the autonomous agent.

In some embodiments, the infrastructure devices may include one or moresensor devices that may be fixedly attached or positioned within an(driving) environment, such that a fixed (or substantially) coordinate(geographic) location of the one or more infrastructure sensor devicesmay be known. Accordingly, such fixedly arranged infrastructure devicesmay have a fixed field-of-detection. For instance, a camera fixed in adriving environment may have a fixed field-of-view. In some embodiments,the infrastructure devices may include one or more sensors devices thatare movably positioned within an environment, such that a coordinatelocation of the one or more sensor devices varies. In such embodiments,the infrastructure devices may have a variable field-of-detection andmay be capable of sensing data along multiple trajectories within anenvironment.

In a first implementation, S210 may function to automatically collectstreams of data from one or more infrastructure devices that are incommunication proximity (e.g., a predetermined distance that enablesshort-range communication) of the autonomous agent. In some embodiments,the infrastructure devices may be configured to communicate with anautonomous agent using short-ranged communication schemes or systems. Insuch embodiments, once the autonomous agent has entered (or traveledinto) a communication range or proximity of a given infrastructuredevice, the autonomous agent may function to automatically detectsignals from the infrastructure device and automatically collect dataoriginating from the infrastructure device.

In a second implementation, S210 may function to automatically collectstreams of data from one or more infrastructure devices that are apredetermined distance from an operating autonomous agent. That is, insome embodiments, an operating environment of an autonomous agent mayinclude a plurality of infrastructure devices, however, the autonomousagent may be configured to automatically collect data from only a subsetof the plurality of infrastructure device within the predetermineddistance of the autonomous agent and possibly, ignore data incoming fromother infrastructure devices outside of the predetermined distance ofthe autonomous agent. In this way, the autonomous agent may function tocollect data having a more immediate or higher relative importance forpending and/or immediate operating decisions.

In a variant of the second implementation, S210 may function toautomatically collect streams of data from one or more infrastructuredevices that are a predetermined distance of the autonomous agent andweigh or consider data collected from the one or more infrastructuredevices within an active trajectory or travel path of the autonomousagent differently than data from infrastructure devices that are not orno longer within a trajectory or travel path of the autonomous agent.That is, in some embodiments, S210 may function to weigh data frominfrastructure devices that are substantially coincident with a positionof the autonomous agent and ahead of a travel path of the autonomousagent with additional (increased) weight than a weight afforded to datafrom infrastructure devices behind or that has been passed by theautonomous agent along its travel path.

In some embodiments, the data collected from the one or moreinfrastructure devices may include compressed and/or semantically densedata regarding one or more features of an environment. In someembodiments, the field-of-detection of given infrastructure devicecomprises a geometrically defined region and within the geometricallydefined region, the infrastructure device may be configured to sense orcollect a semantic abstraction (e.g., a general shape or size,positions, velocity (moving or not moving) of the features, objects,and/or agents within the geometrically defined region.

Additionally, or alternatively, in some embodiments, the one or moreinfrastructure devices may be configured to sense or detect data withinthe geometrically defined region and derive and/or compute a state dataabout the circumstances within the geometrically defined shape. Forinstance, if the geometrically-defined shape or sensing region is asquare that includes a sidewalk or similar pedestrian path, a firstinfrastructure sensor device may function to identify whether there arestatic (objects or persons that are not moving) and/or dynamic agents(objects or persons that are moving) on the sidewalk and provide asstate data to the autonomous agent an indication confirming staticagents or dynamic agents positioned on the sidewalk (e.g., coordinatedata of the static or dynamic agents or the like operating within ageographic location). In some cases, if there are no agents positionedwithin the geometrically-defined sensing region, the state data may bean indication of no agents (e.g., “Clear”). Thus, in such embodiments,rather than sending a full representation of a scene within thegeometrically-defined shape or sensing region, the infrastructure devicemay provide semantically dense state data to an autonomous agent. As afew examples of state data, the infrastructure sensing devices mayindicate Agent or No Agent, Static Agent or Dynamic Agent, Clear or NotClear, Busy (Active) or Not Busy (Not Active), and/or any suitablesimplified and/or derivative information about circumstances within asensing region of an infrastructure device that may be provided to anautonomous agent.

S215, which includes building a data buffer comprising decisioning data,may function to arrange and/or store streams of data from one or moredata sources within a data buffer, which may sometimes be referred toherein as a historical and/or global data buffer. The data bufferpreferably functions to store historical data as well as presentlycollected data (e.g., real-time data or near real-time). In someembodiments, S215 may function to collect data from devices and/orcomponents of the autonomous agent, onboard sensors of the autonomousagent, and/or data from one or more infrastructure devices. Thus, in apreferred embodiment, the buffer may be considered a global data bufferor the like that functions to collect data from the various sensingdevices onboard an autonomous agent as well as sensing data frominfrastructure devices not onboard the autonomous agent, which mayadditionally include data from other remote data sources (e.g.,Internet-based data feeds) or the like.

S215 may function to build the data buffer by sequentially storing datareceived (in S210) within the data buffer based on a time at which thedata was received or collected by an autonomous agent. However, astimestamp data that may be appended with the one or more streams of datacollected by an autonomous agent may be analyzed, S215, may function tocontinuously and/or periodically reconfigure the data buffer using aglobal time synchronization technique in which the data elements withinthe data buffer are adjusted and/or re-ordered according to thetimestamp data associated with each of the one or more streams of data.

Accordingly, the data buffer may function to operate in dual states inwhich a first portion of data comprising historical data stored in thedata buffer may be globally time synchronized and a second portion ofdata comprising recently (within 0 to 15 seconds) stored data and/orreal-time data may be data that is not yet globally time synchronized.It shall be noted that recently stored data may be considered data thathas not been stored beyond a predetermined threshold of time and/or datathat has been stored but not yet globally time synchronized with thepreviously stored data within the data buffer.

In a preferred implementation, S215 implements a global timesynchronization module that takes into account latencies in thecommunication and/or collection of the one or more streams of data andfunctions to synchronize the one or more streams of data collected fromall data sources. In some embodiments, a latency of a data source may beknown based on a communication method and/or global position system(GPS) based position of the data source relative to an autonomous agentat the time of receipt of the stream of data (alternatively, at the timeof transmitting the stream of data). With respect to infrastructuredevices, because the infrastructure devices are positioned away from anautonomous agent in the circumstances and/or environment surrounding theautonomous agent, the data collected, sensed, and/or derived by theinfrastructure devices are generally communicated to an autonomous agentover using some wireless communication system (e.g., short-wavecommunication) or the like. Accordingly, regarding each infrastructuredevice that communicates data to an autonomous agent, S215 may functionto calculate or estimate a latency value in communication between eachrespective infrastructure device and the autonomous agent. S215 mayfunction to use the estimated or calculated latency value to adjust anactual event time and/or receipt time for the data stream associatedwith each infrastructure device that communicates data to an autonomousagent. For instance, each data stream may be appended with metadata(e.g., a timestamp) identifying a time of receipt by the autonomousagent and S215 may function to reduce or subtract from the time ofreceipt for each stream of data the latency value according to thestream of data's data source (e.g., infrastructure device). That is, alatency value may be computed based on a timestamp provided theinfrastructure device indicating a time at which the data was sensed,obtained, and/or originated by the infrastructure device. S215 mayfunction to compute the latency value based on a difference between thetimestamp data and a second timestamp by the autonomous agent indicatinga time at which the data from the infrastructure data was received.

In one variation of S215, a latency value is predetermined or known inadvance (rather than calculated or estimated by the autonomous agent)for each type of infrastructure device and the predetermined and/orknown latency value for each type of infrastructure device may bereduced or deducted from a time of receipt of each stream of data todetermine the globally synchronized time for the stream of dataassociated with a given infrastructure device.

In a second implementation, S215 may function to globally timesynchronize the multiple streams of data generated by each of the datasources, including infrastructure devices based on timestamp data. Thatis, in this second implementation, each of the data sources may functionto operate on a common clock or a same synchronized clock with theautonomous agent. Accordingly, at a time of recording and/or sensingdata, each of the data sources, such as one of the infrastructuredevices, may function to record a timestamp based on the common clock orthe synchronized clock indicating a synchronized time at which datawithin the data streams was actually observed by the onboard sensors andthe offboard infrastructure devices rather than a time at which the datastreams from each of the onboard sensors and the offboard infrastructuredevices were received or collected by the autonomous agent. S215 mayfunction to adjust a serial order of the data streams received from theone or more data sources using the timestamp data in the placed of atimestamp associated with a receipt of the streams of data at anautonomous agent.

S215, according to one or more of the above-described global timesynchronization schemes, preferably functions to re-order and/orrearrange a sequential order of the one or more data streams within theglobal data buffer. In such embodiments, S215 preferably functions tore-order and/or rearrange data elements of the data streams from thedata sources from a first (or initial) position within the global databuffer to a second (or subsequent) position within the global databuffer based on a global synchronized time for each of the dataelements. Thus, S215 may function to interject or reposition dataelements from streams of data originating with the one or moreinfrastructure devices into a time synchronized position in the globaldata buffer to increase an accuracy of the decisioning data to theautonomous agent.

S220, which includes processing the collected decisioning data, mayfunction to perform one or more data processing techniques against thedecisioning data to derive decisioning inputs for selecting behavioralpolicy by the autonomous agent. In one embodiment, S220 includesimplementing one or more data fusion techniques (S222) and generatingone or more hypothesis regarding a (current or present) state of one ormore agents within an environment of the autonomous agent (S224).

Agent Tracking

S221, which includes tracking one or more agents and/or objects, mayfunction to track one or more agents and/or objects that may beidentified by the autonomous agent and/or infrastructure devices. In oneor more embodiments, S221 may function to identify one or more agentsand/or objects in circumstances surrounding one or more of theautonomous agent and the infrastructure devices based on the collecteddecisioning data. In such embodiments, the collected decisioning datamay include sensor data from one or more of the autonomous agent and theinfrastructures devices. Accordingly, the one or more agents beingtracked may include agents identified based on the sensor data of theautonomous agent and/or sensor data obtained by the infrastructuredevices.

In some embodiments, S221 may function to track the one or more agentsvia one or more of the autonomous agent, infrastructure devices, and/ora remote autonomous agent service (e.g., a cloud-based server (adistributed computing network or the like)). That is, in someembodiments, each of the autonomous agent, infrastructure devices,and/or the autonomous agent service may function to perform a trackingfunctionality of agents identified in circumstances surrounding theautonomous agent and/or the infrastructure devices. Thus, it shall benoted that, a tracking functionality described herein may be performedby or at any of the autonomous agent, the infrastructure devices, andthe autonomous agent service. In some embodiments in which the trackingfunctionality of an agent and/or an object is performed remote from theautonomous agent, possibly at the infrastructure devices and/or theremote autonomous agent service, the resultant tracking data may betransmitted to the autonomous agent from the tracking source.

In a preferred embodiment, S221 may function to identify and/or insert adedicated track (i.e., a tracklet) for sensor data obtained from eachrespective sensor of the autonomous agent and/or the infrastructuredevices. That is, in such embodiments, S221 may insert sensor data fromeach distinct sensor of the autonomous agent and/or the infrastructuresdevices into a dedicated memory section and/or independent memorysections of the decisioning data buffer and/or similar data buffer. Inthis way, S221 may function to track agents of each distinct sensorindependently.

Additionally, or alternatively, in some embodiments, S221 may functionto merge the distinct tracklets for each sensor (e.g., sensor tracklet)into a master tracker. The master tracker, in some embodiments, mayinclude a convergence of all decisioning data and/or sensor data fromwhich one or more hypothesis regarding a tracking and/or estimatedtrajectory of one or more agents may be computed. The convergence of thedistinct tracklets into the master tracker may enable S221 to build acomposite of fused tracking from multiple perspectives of each distinctagent and/or object that is identified in the sensor data. That is, iftwo or more distinct sensors having an overlapping detection and/ortracking of a subject agent, the sensor data (i.e., perspectives) of thetwo or more distinct sensors may be combined to build a composite orcomprehensive tracking of the subject agent. For instance, a firstcamera of the autonomous agent may function to sense and track a subjectagent (e.g., a pedestrian) and second camera of an infrastructure devicemay function to sense and track the subject agent from a distinctperspective. In such instance, S221 may function to combine the videodata of the first and second camera to build a composite tracking of thesubject agent. In some embodiments, the composite tracking of a subjectagent and/or object may be used as input for estimating a trajectoryand/or an estimated behavioral policy for the subject agent.

Agent Classification

S222, which includes identifying one or more agents and/or objects, mayfunction to classify each of the one or more agents and/or objectsdetected within the circumstances surrounding the autonomous agent. In apreferred embodiment, S222 may function to classify and/or categorizeeach identified agent and/or object within circumstances surrounding oneor more of the autonomous agent and/or the infrastructure device.

In some embodiments, S222 may function to function to perform featureextraction of an agent or an object identified based on sensor data andperform a classification of the agent and/or the object based on afeature extraction dataset. In such embodiments, S222 may function toimplement any suitable feature extractor include a deep machine learningmodel or the like. It shall be noted that S222 may function to implementany suitable object and/or agent classification technique,classification algorithm (e.g., trained machine learning-basedclassification algorithms, classification models, etc.), statisticalclassification models, and/or the like for purposes of classifying anagent and/or an object.

Fusion

S223, which includes implementing one or more data fusion techniques,functions to collect or aggregate streams of data from the onboardand/or offboard sensors or data sources and provide the data to a fusionsystem or fusion module to generate one or more estimations and/orclassifications regarding the autonomous agent and the features (e.g.,agents, objects, areas, etc.) in an operating environment of theautonomous agent.

In a first implementation, S223 may function to provide as input into adata fusion system the onboard sensor data separately from the offboardsensor data. That is, in this first implementation, sensor datacollected from some or all of the onboard data sources of an autonomousagent may be aggregated and synthesized using the fusion system of theautonomous vehicle independently from a synthesis of the offboard sensordata or the like. A proximity of onboard data sources to the fusionsystem of the autonomous agent may enable an efficient passing ofonboard sensor data from the onboard data sources to a processing systemof the fusion system whereas a latency associated with sensor dataexchanged between the autonomous agent and the one or moreinfrastructure devices and/or remote data sources includes a timingoffset or timing misalignment that limits an ability of the fusionsystem to process the onboard and offboard sensor data together.

Additionally, and/or alternatively, in this first implementation, theonboard sensor data and the offboard sensor data may be provided asinput into a common fusion system or module. Additionally, oralternatively, S223 may provide the onboard sensor data as input into afirst fusion system and the offboard sensor data as input into a secondfusion system. In such embodiment, the first fusion system may bespecifically configured to process onboard sensor data and the secondfusion system may be specifically configured to process offboard sensordata.

In a second implementation, S223 may function to process the onboardsensor data and the offboard sensor data together (e.g., at the sametime or substantially the same time) using a common fusion system. Thatis, in this second implementation, it may be possible to pre-process theonboard sensor data and the offboard sensor data to time synchronize thedata streams originating from the onboard data sources and the offboarddata sources to reduce or eliminate discrepancies in synchronizationresulting from communication latencies between the autonomous agent andthe one or more offboard infrastructure devices and/or one or moreoffboard data sources (e.g., cloud-based or Internet-based data feeds,communications from other autonomous agents (vehicle-to-vehiclecommunications), and/or the like). Accordingly, the disparate datastreams from the onboard data sources (sensors) and the disparate datastreams from the offboard data sources (infrastructure devices, distinctautonomous agents, and the like) may be combined into a unified (orsingle) data stream and provided into the fusion system for processing.

Hypotheses/Intent Estimation

S224, which includes generating one or more hypotheses regarding a(current or present) state of one or more agents within an environmentof the autonomous agent and one or more hypotheses regarding a state ofthe autonomous agent, may function to output from the fusion systemand/or associated data processing module a hypothesis for each of theautonomous agent and identified agents external to the autonomous agent.Preferably, the hypothesis for each external (environmental agent) andautonomous agent includes an estimation of a geographic position (orthree-dimensional coordinate or the like), a velocity, and/oracceleration. Additionally, or alternatively, the hypothesis forexternal agents may include shape estimations or descriptions and/orclassifications having probabilistic values with variances andcovariances. The classification outputs preferably includes aclassification of whether an identified agent within a scene oroperating environment of the autonomous agent is static or dynamic andthe type or class of agent (e.g., a person, a vehicle, a bicyclist, ananimal, a stationary object, etc.). The fusion system may function toimplement one or more and/or a combination of state estimation and/orclassification algorithms and models including simple gaussianfunctions, predictive or inferential machine learning models, machinelearning classifiers, and the like.

S230, generating an inference and/or identifying an intent of each ofthe agents in circumstances surrounding the autonomous agent and/orsurrounding the infrastructure devices, may function to receive as inputone or more of the hypotheses generated in S224 to generate one or morepotential intents for agents in the circumstances as well as an intentestimation for the autonomous agent. An intent of an agent preferablyrelates to an estimation of a behavioral policy (i.e., an estimation ofwhat an agent is expected to do, one or more expected or future actions,or the like) having a highest probability of being executed by a givenagent. The intent for each agent is preferably estimated using amulti-policy decision-making module that functions to estimatebehavioral policy for each agent based on generated hypotheses data(derived in S220/S224) and data from the global data buffer (e.g.,historical data buffer) preferably in a globally time synchronizedstate.

It shall be noted that intent estimation may preferably be performedonboard an autonomous agent, intent estimation may be performed by anyand/or a combination of the autonomous agent, an infrastructure device,a remote autonomous agent service and/or the like. In one exampleindirect intent estimation (e.g., intent estimation by an agent otherthan a subject autonomous agent), sensor data may be collected at aninfrastructure device and transmitted to a remote autonomous agentservice implemented via a cloud service or the like. The remoteautonomous agent service may function to compute intent estimations foreach of the agents identified by the infrastructure device.Additionally, or alternatively, the infrastructure, itself, may functionto compute intent estimations based on sensor data or the like obtainedby the infrastructure device.

In a preferred embodiment, S230 may function to identify agentsoperating in the environment and a static or dynamic classification ofeach agent. In such embodiments, S230 may function to selectivelyprovide as input into the multi-policy decision-making module only thoseagents identified or classified as being dynamic (moving or active)agents in the operating environment of the autonomous agent therebylimiting intent computations by the multi-policy decision-making moduleto the dynamic agents. A technical benefit of the selective input ofdynamic agents into the multi-policy decision-making module is that thistechnique may function to preserve (the limited) computational resourcesof the autonomous agent, which enables the autonomous agent to computewith higher efficiencies and speed intents of actors or agents in theenvironment having the highest impact on behavioral policy selection bythe autonomous vehicle.

Preferably, the multi-policy decision-making module includes a simulatoror similar machine or system that functions to estimate future (i.e.,steps forward in time) behavioral policies (operations or actions) foreach of the agents identified in an operating environment of theautonomous agent including potential behavioral policies that may beexecuted by the autonomous agent, as described in U.S. patentapplication Ser. No. 14/814,766, which is incorporated in its entiretyby this reference. The simulations may be based on a current state ofeach agent (e.g., the current hypotheses) and historical actions orhistorical behaviors of each of the agents derived from the historicaldata buffer (preferably including data up to a present moment). Thesimulations may provide data relating to interactions (e.g., relativepositions, relative velocities, relative accelerations, etc.) betweenprojected behavioral policies of each agent and the one or morepotential behavioral policies that may be executed by the autonomousagent.

Policy Enumeration

Based on the forward simulations by the multi-policy decision-makingmodule, S240, which includes enumerating identifying potentialbehavioral policies for execution by the autonomous agent, preferablyfunctions to output a plurality of potential behavioral policies havinghighest probabilities of being executed by the autonomous agent, asshown by way of example in FIG. 3. In some embodiments, S240 mayfunction to output potential behavioral policies that may be most safelyexecuted by the autonomous agent. In one embodiment, S240 may functionto output potential behavioral policies that optimizes over a pluralityof operating factors including safety and efficiency of operating theautonomous agent, which may include limiting a disturbance of theautonomous agent within the operating environment.

It shall be noted that the universe of behavioral policies and/oravailable behavioral policies that may be considered by the multi-policydecision-making module may be based on a plurality of distinctbehavioral policy sources including, but not limited to, a remoteoperator of the autonomous agent or a remote autonomous agent service, apredetermined set of policies, a geographical map or route mappingaugmented with select behavioral policies available for execution alongdistinct geographic locations within the geographical map or routemapping, user/passenger preferences, and the like. Accordingly, anysuitable behavioral policy source may function to populate a behavioralpolicy database that may include all possible behavioral policies for agiven autonomous agent.

Additionally, or alternatively, S240 may function to delimit thepotential behavioral policies for execution by the autonomous agentbased on one or more predetermined thresholds relating to probabilitiesof execution by the autonomous agent. That is, in some embodiments, S230may function to generate hundreds and if not, thousands of simulationsresulting in hundreds or thousands of potential behavioral policies forexecution by the autonomous agent in a given circumstance. Therefore,S240 may function to identify only a subset of those generatedbehavioral policies according to predetermined threshold identifying oneor more minimum probability values for safely executing an action or anoperation by the autonomous agent or one or more minimum probabilitiesvalue for successfully executing an operation or an action by theautonomous agent in a given circumstance or real-time scenario.

S250, which includes selecting one of the plurality of behavioralpolicies, functions to select one of the potential behavioral policiesbased on one or more predetermined or dynamic selection criteria. Theselection criteria may be based on any suitable behavioral policyselection factors that may be delineated in advance of operating theautonomous agent or dynamically based on one or more features relatingto an operating environment or operating mode of the autonomous agent.For instance, the selection criteria may be predetermined and/or setsuch that the autonomous agent functions to select the behavioral policywith a highest probability of being executed safely. In another example,if an operating circumstance of the autonomous vehicle includes anemergency situation, the selection criteria may be dynamic and set suchthat the autonomous agent functions to select a behavioral policy fromthe tractable set of behavioral policies that requires a (weighted)balance between efficiency in operation and safety or the like.

In some embodiments, the set of behavioral policies available forselection by the autonomous agent may be further delimited based on datacollected from the one or more infrastructure devices. For instance, insome embodiments, semantic classification or other classification datarelating to one or more features of the environment may additionally beused as input in selecting an optimal behavioral policy in S250.Accordingly, for each behavioral policy of the delimited set ofbehavioral policies available for selection by the autonomous agent,S250 may function to data from one or more of the infrastructure devicethat may function to validate or invalidate operations that may beperformed or actions that may be taken by the autonomous agent accordingto a given behavioral policy. For instance, S250 may function toidentify semantic data from an infrastructure device, such as a cameraor a radar installed in the environment, indicating that a sidewalk in apotential turning direction of the autonomous agent is “clear” (i.e., noactive or dynamic agents on sidewalk). This example semantic data mayprovide a negative observation (e.g., a lack of agents in an observablearea or the like) inference within a sensing region or within anobservable scene that validates or confirms that a turn by theautonomous agent can be safely performed in a given circumstance.Correspondingly, semantic data from one or more infrastructure devicesmay provide positive observation data, such as multiple active agents ona sidewalk in a possible turning direction of autonomous agent, mayinvalidate several behavioral policy that requires a turn in thedirection of the sidewalk. Thus, semantic data and/or other relevantinfrastructure sensor data may be re-applied at the behavioral policydecisioning or selection step to further delimit and/or inform a mostoptimal selection of a behavioral policy by the autonomous agent.

Additionally, or preferably, S250 may function to identifyinfrastructure sensing data that relates to one or more imperceptibleregions of an operating environment of an autonomous agent and use theimperceptible offboard sensing data to inform a selection of behavioralpolicy by the autonomous agent. For example, if an autonomous agent(e.g., autonomous vehicle) is positioned behind a stationary object,such as a large bus, on a single lane street, S250 may function toidentify that the autonomous agent cannot perceive whether there isoncoming traffic in an adjacent lane with opposing traffic, as shown byway of example in FIG. 4. In such example, S250 may function to identifyinfrastructure sensing data within the sensing regions that areimperceptible to the autonomous agent that may inform whether abehavioral policy for traversing into the adjacent lane for opposingtraffic may be performed safely by the autonomous agent for travelingpast the large bus to achieve a routing goal, etc.

In response to selecting a behavioral policy, S255, which includesexecuting a selected behavioral policy, may function to automaticallyexecute the selected behavioral policy, as prescribed, by the autonomousagent. Accordingly, the autonomous agent may be controlled and/oroperated based on an execution by an onboard computer or the like ofinstructions associated with the selected behavioral policy.

The systems and methods of the preferred embodiments and variationsthereof can be embodied and/or implemented at least in part as a machineconfigured to receive a computer-readable medium storingcomputer-readable instructions. The instructions are preferably executedby computer-executable components preferably integrated with the systemand one or more portions of the processors and/or the controllers. Thecomputer-readable medium can be stored on any suitable computer-readablemedia such as RAMs, ROMs, flash memory, EEPROMs, optical devices (CD orDVD), hard drives, floppy drives, or any suitable device. Thecomputer-executable component is preferably a general or applicationspecific processor, but any suitable dedicated hardware orhardware/firmware combination device can alternatively or additionallyexecute the instructions.

Although omitted for conciseness, the preferred embodiments includeevery combination and permutation of the implementations of the systemsand methods described herein.

As a person skilled in the art will recognize from the previous detaileddescription and from the figures and claims, modifications and changescan be made to the preferred embodiments of the invention withoutdeparting from the scope of this invention defined in the followingclaims.

What is claimed is:
 1. A system for intelligently implementing anautonomous agent, the system comprising: a plurality of offboardinfrastructure devices arranged geographically remote an autonomousagent and that: collect observation data relating to circumstancessurrounding a travel route of the autonomous agent; the autonomous agentcomprising: a communication interface that enables the autonomous agentto communicate with each of the plurality of offboard infrastructuredevices; an onboard sensor suite comprising a plurality of distinctsensors arranged on the autonomous agent and that collect observationdata relating to circumstances surrounding the autonomous agent from aperspective that is distinct a perspective of the plurality of offboardinfrastructure devices; a decisioning data buffer storing at least afirst stream of observation data from the onboard sensor suite and asecond stream of observation data from the plurality of offboardinfrastructure devices; one or more onboard computing devices that:implements a tracking of one or more agents within the circumstancessurrounding the autonomous agent based on the first stream of data andthe second stream of data; computes an intent estimation for each of theone or more agents based on the first stream of data and the secondstream of data; computes state data for each of the one or more agentsand for the autonomous agent; implements a multi-policy decision-makingmodule that identifies a plurality of candidate behavioral policies forthe autonomous agent based on the tracking and the intent estimation;selects one of the plurality of candidate behavioral policies andexecutes the selected one of the plurality of candidate behavioralpolicies for controlling an autonomous operation of the autonomousagent; and a remote autonomous agent service being implemented by adistributed network of computing devices and that is in operablecommunication with each of the autonomous agent and each of theplurality of offboard infrastructure devices, wherein the remoteautonomous agent service computes one or more: the tracking of one ormore agents within the circumstances surrounding the autonomous agentbased on the first stream of data and the second stream of data; theintent estimation for each of the one or more agents based on the firststream of data and the second stream of data.
 2. The system according toclaim 1, wherein the one or more onboard computing devices furtherimplement: a time synchronization module that synchronizes the firststream of observation data and the second stream of observation data tothe autonomous agent by synchronizing: the first stream of observationdata to the autonomous agent based on a first computed communicationlatency between the onboard sensor suite and the autonomous agent; thesecond stream of observation data to the autonomous agent based on asecond computed communication latency between the plurality of offboardinfrastructure devices and the autonomous agent.
 3. The system accordingto claim 2, wherein the one or more onboard computing devices further:rearrange a position of the first stream of observation data from afirst position to a second position within the decisioning data bufferbased on the synchronization; and rearrange a position of the secondstream of observation data from a first position to a second positionwithin the decisioning data buffer based on the synchronization.
 4. Thesystem according to claim 1, wherein the one or more onboard computingdevices further implement: a synchronization module that synchronizesthe first stream of observation data of the autonomous agent and thesecond stream of observation data from offboard infrastructure devicesaccording to a common clock of the autonomous agent.
 5. The systemaccording to claim 4, wherein the one or more onboard computing devices:in response to the synchronization of the first stream of observationdata and the second stream of observation data, data from the firststream of observation data and data from the second stream ofobservation data are repositioned to a historical position within thedecisioning data buffer that is associated with an earlier point in timerelative to data that is positioned beyond the data from the firststream and the second stream of observation data.
 6. The systemaccording to claim 1, wherein the one or more onboard computing devices:stores data obtained from each of the plurality of offboardinfrastructure devices and data obtained from each of the plurality ofdistinct sensors of the onboard sensor suite within a distinct track ofmemory that is independent from other tracks of memory, wherein anintent estimation for any agents identified within the data obtainedfrom each of the plurality of offboard infrastructure devices and thedata obtained from each of the plurality of distinct sensors is computedbased on the distinct track of memory for the respective offboardinfrastructure device or the respective distinct sensor.
 7. The systemaccording to claim 6, wherein: the distinct track of memory for the dataobtained from each of the plurality of offboard infrastructure devicesand the data obtained from each of the plurality of distinct sensors iscombined into a master track of memory; the intent estimation for eachof the one or more agents is based on the master track of memory.
 8. Amethod for autonomous decisioning and operation by an autonomous agent,the method comprising: collecting decisioning data including: collectinga first stream of data comprising observation data obtained by one ormore onboard sensors of the autonomous agent, wherein each of the one ormore onboard sensors is physically arranged on the autonomous agent;collecting a second stream of data comprising observation data obtainedby one or more offboard infrastructure devices, the one or more offboardinfrastructure devices being arranged geographically remote from and inan operating environment of the autonomous agent, the operatingenvironment of the autonomous agent including a predetermined radiusfrom a geographical position of the autonomous agent while operatingalong a structured or an unstructured route of the autonomous agent;implementing a decisioning data buffer that includes the first stream ofdata from the one or more onboard sensors and the second stream of datafrom the offboard sensors; generating current state data; generatingintent data for each of one or more agents within the operatingenvironment of the autonomous agent; identifying a plurality ofcandidate behavioral policies; and selecting and executing at least oneof the plurality of candidate behavioral policies.
 9. The methodaccording to claim 8, wherein: the first stream of data includes datarelating to sensed observations of circumstances surrounding theautonomous agent obtained from a perspective of the autonomous agent byeach of the one or more onboard sensors; the second stream of dataincludes data relating to sensed observations of circumstances withinthe operating environment of the autonomous agent obtained from anexternal perspective toward a route of the autonomous agent that is madeby each of the offboard infrastructure devices.
 10. The method accordingto claim 9, wherein the second stream of data includes data relating toan operating state of at least one of the one or more offboardinfrastructure devices.
 11. The method according to claim 8, wherein inresponse to traveling into a communication proximity of at least one ofthe one or more offboard infrastructure devices, automaticallycollecting the second stream of data from the at least one or moreoffboard infrastructure devices within the communication proximity. 12.The method according to claim 8, wherein: a field-of-sensing of the oneor more offboard infrastructure devices comprises a geometricallydefined region, the one or more offboard infrastructure devices may beconfigured to sense or collect semantic abstractions of objects withinthe geometrically defined region.
 13. The method according to claim 8,wherein: the second stream of data from the offboard infrastructuredevices includes semantically dense state data of a scene.
 14. Themethod according to claim 8, wherein implementing the decisioning databuffer includes: sequentially storing data received from the firststream of data and the second stream of data based on a time at whicheach of the first stream of data and the second stream of data wasreceived.
 15. The method according to claim 8, wherein implementing thedecisioning data buffer further includes: computing a global timesynchronization between the first stream of data and the second streamof data based on timestamp data appended with the second stream of dataprovided by the one or more offboard infrastructure device, whereincomputing the global time synchronization includes computing a latencyvalue for the second stream of data based on calculating a differencebetween a first timestamp data indicating a first time at which thesecond stream of data was obtained by the one or more offboardinfrastructure devices and a second timestamp data indicating a secondtime at which the second stream of data was collected by the autonomousagent.
 16. The method according to claim 15, wherein implementing thedecisioning data buffer includes: repositioning within the decisioningdata buffer data from the second data stream based on the computedlatency value, wherein the repositioning includes moving the data fromthe second data stream from a first position to a second position withinthe decisioning data buffer that is earlier in historical time.
 17. Amethod for autonomous decisioning and operation by an autonomous agent,the method comprising: collecting decisioning data including: collectinga first stream of data comprising observation data obtained by one ormore onboard sensors of the autonomous agent, wherein each of the one ormore onboard sensors is physically arranged on the autonomous agent;collecting a second stream of data comprising observation data obtainedby one or more offboard infrastructure devices, the one or more offboardinfrastructure devices being arranged geographically remote from and inan operating environment of the autonomous agent, wherein afield-of-sensing of the one or more offboard infrastructure devicescomprises a geometrically defined region, and wherein the one or moreoffboard infrastructure devices may be configured to sense or collectsemantic abstractions of objects within the geometrically definedregion; implementing a decisioning data buffer that includes the firststream of data from the one or more onboard sensors and the secondstream of data from the offboard sensors; generating current state data;generating intent data for each of one or more agents within theoperating environment of the autonomous agent; identifying a pluralityof candidate behavioral policies; and selecting and executing at leastone of the plurality of candidate behavioral policies.
 18. The methodaccording to claim 17, wherein: the first stream of data includes datarelating to sensed observations of circumstances surrounding theautonomous agent obtained from a perspective of the autonomous agent byeach of the one or more onboard sensors; the second stream of dataincludes data relating to sensed observations of circumstances withinthe operating environment of the autonomous agent obtained from anexternal perspective toward a route of the autonomous agent that is madeby each of the offboard infrastructure devices.
 19. The method accordingto claim 18, wherein: the second stream of data includes data relatingto an operating state of at least one of the one or more offboardinfrastructure devices.
 20. The method according to claim 17, wherein:the second stream of data from the offboard infrastructure devicesincludes semantically dense state data of a scene.